Method for segmenting an image

ABSTRACT

A method for isolating an element of an image made up of pixels comprising the steps of classifying the pixels into different groups based on the color value of the pixel, blurring the image, locating a pixel in the blurred image that has a predetermined color value corresponding to the element to be isolated, and growing a mask from the located pixel.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Provisional PatentApplication Serial No. 60/368,472 filed Mar. 28, 2002.

FIELD OF INVENTION

[0002] This invention relates to a method to identify and isolate acomponent or feature of a digital image (automated segmentation). Moreparticularly, the invention relates to a method for isolating a featureof an image, such as the teeth, and modifying the isolated feature toshow the anticipated effect of a treatment such as whitening orisolating a feature such as hair and modifying it to show the effect ofa treatment such as coloring.

SUMMARY OF INVENTION

[0003] An image may be captured using any of a variety of methods, butmost typically using a standard image capture device (e.g., a digital orweb camera or a scanned photographic image might be used), and displayed“live” on a screen. In one embodiment of the invention a “targeting”area may be displayed on the screen, which helps standardize the size(distance from camera) and placement of the image. Once the image iscaptured, the software analyzes the image, placing each pixel into acolor category. All pixels in a category will be part of a particularcomponent or feature of the image, thus isolating and identifying thatelement.

[0004] In one embodiment a digital image of a human face is analyzed toidentify pixels that represent the teeth. It identifies the teeth in theimage, and then determines their current color and their likely colorafter a teeth-whitening treatment, which may be determined by alook-up-table or a simple conversion equation.

[0005] In another embodiment a digital image of a human head is analyzedto identify pixels that represent hair. It identifies the hair in theimage, and then determines its current color. Additional software thenuses that information to recommend what coloring products & processes touse to achieve a target color, or to simulate the result when aparticular product & process are applied to the existing hair.

[0006] One manifestation of the invention is a device for capturing animage and locating a feature of the image using a segmentation program.

[0007] Another manifestation of the invention is a device as describedabove wherein the located feature is modified and redisplayed as part ofthe original image.

[0008] Another manifestation of the invention is a segmentation programfor locating a feature of a photographic image.

[0009] A more specific manifestation of the invention is a device forcapturing an image of a facial feature such as the teeth or hair,locating the facial feature using a segmentation program, modifying thefacial feature to display the effect of a change in tooth whiteness or achange in hair color, and displaying the modified image. This device isparticularly useful in selling cosmetics.

[0010] Another more specific manifestation of the invention is asegmentation program in which a K-means algorithm is used to classifythe pixels in an image into color groups.

[0011] Another more specific manifestation of the invention is asegmentation program as described immediately above where the originalimage is blurred to merge smaller segments of the same feature in theimage and a pixel representative of the desired feature is located inthe blurred image.

[0012] Still another manifestation of the invention is a segmentationprogram in which the pixel identified as described above is grown into amask using a connection definition.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013]FIG. 1 is a flow chart illustrating a segmenting process inaccordance with one embodiment of the invention.

[0014]FIG. 2 is a series of images that illustrate the photographiceffect at different steps of a segmentation process in accordance withone embodiment of the invention. FIG. 2(a) is an image prior toprocessing; FIG. 2(b) illustrates the image after pixel classificationby a K-means algorithm; FIG. 2(c) illustrates the effect of blurring theclassified image; FIG. 2(d) illustrates identification of target pixel;FIG. 2(e) is an image of the mask formed using a connection definition,FIG. 2(f) of the segmented feature.

[0015]FIG. 3 is an apparatus in accordance with one embodiment of theinvention.

DETAILED DESCRIPTION

[0016] As illustrated in FIG. 1 to conduct the segmentation process thecaptured image is converted from RGB values to a colorimetric space suchas CIELAB. If the RGB values are already in a standard color space (e.g.sRGB (See M. Anderson et. al. , “Proposal for a Standard Default ColorSpace for the Internet,” IS& /SID 4th Color Imaging Conference, pp.238-246, 1996.)), then the transformation is well known andstraight-forward. If the RGB values are not in a standard color space,then it is necessary to determine a transformation that will convert theRGB values to CIELAB in a manner that is known in the art. Thistransformation can be determined once for the capture device that isused, prior to the on-going operation of the device. Details of thisoperation are given in the block diagram, FIG. 1, and the discussionbelow.

[0017] Once, the image is in the CIELAB color space, each pixel in theimage is classified (Box B). The method for performing this operation isthe K-Means algorithm. For a discussion of this algorithm see C.Therrien, Decision, Estimation, and Classification, John Wiley & Sons,NY, 1989, pp 217-218. K-Means is a classic iterative pattern recognitionalgorithm in which a set of data (i.e. the individual pixels) isoptimally divided into a set of K classes (i.e. color groups). In thismethod, an optimization problem is solved to determine the class inwhich a pixel is contained (i.e. into which color group does the pixelbest fit). The selection of the number of categories into which thepixels are classified depends upon the number of distinguishablefeatures in the image or the portion of the image that is the object ofthe segmentation routine. For the case of teeth, the features may beskin, lips, and teeth (in this case K=3). Additional facial featuresthat are spatially disjointed from the teeth (e.g. moustaches) can beeither classified as teeth or non-teeth (e.g., skin or lips) groups. Ifthe feature is incorrectly classified as teeth, the algorithm will keepthe teeth and the additional facial feature separate due to theirunconnectedness or separation in the image. FIG. 2b illustrates theoutput of a K-means algorithm (K=5).

[0018] After executing the K-means algorithm, there may be severaldiscontinuous or disjointed segments that are contained in the sameclass or color group. For example teeth and the specular reflection fromglossy lips may be within the same class due to the fact that both willappear close to white. If one is only interested in modifying oradjusting the image of the teeth, then it is necessary to separate theteeth pixels from the lip-gloss pixels. If these two sections are notconnected, then this separation can be achieved by first identifying apixel that is within the desired section such as the teeth area. In oneembodiment, assuming the desired feature represents the largest sectionof the image or the targeted area of the image areawise, thisidentification is achieved by performing a blurring operation on theoriginal RGB image (Box C). The blurred image represents an averaging ofthe local pixels. Blurring insures that the pixel selected in Box D thatis closest in color to the expected color of the desired feature will bein an area of the image that corresponds to the segmented or isolatedfeature and that the pixel selected is not an aberrant pixel in an areaof the image that is not the element to be isolated. The blurringoperation has the effect of smearing out the segments that are smallerthan the blurring kernel, to the point where no feature including thesegment's color is visible. For segments that are larger then the sizeof the blurring kernel, the color of the segment will remain enablingthe determination of the location of one pixel in that segment. The sizeof the blurring kernel is selected to be smaller than the desiredfeature. To identify teeth, the expected value might be the whitestpixel. To identify the hair, the expected value might be determinedempirically by collecting a set of hair images, blurring them, andcomputing the average pixel values across the set of images.

[0019] The picture shown in FIG. 2 has several regions that are the samecolor as the element that is the target of the isolation. The desiredtarget is the largest region. To identify a pixel in this region, theportion of the image in the boxed target area is blurred as shown inFIG. 2c using a blurring kernal that is smaller than the desired target.This is the output from Box C in FIG. 1. Assuming for explanation, thatthis region is green, as the next step in the process, the greenestpixel in the blurred image is selected. This pixel location is outputfrom Box D in FIG. 1.

[0020] Having identified the location of one pixel that is within thesegment of the class to be separated from the image, to identify therest of the desired feature, a mask is grown by adding to the identifiedpixel all pixels that are in this category and connected using aconnection definition such as an 8-point connection definition which iswell known in the art (Box E). The implementation of the growthalgorithm is such that it is relatively insensitive to the connectiondefinition. The details of this growth algorithm are given in the blockdiagram discussion. This mask is the output of Box E and is illustratedby the image shown in FIG. 2e.

[0021] The mask defines the area of interest. If the mask reaches any ofthe edges of the image or is outside of some defined region, then thearea of interest was either not found or not entirely within the image.In this case, the user may be instructed to relocate the desired target(e.g., teeth) in the target box 18 as shown in FIG. 3 and discussedbelow (Box F and Box H). If the mask is good, then the image in the maskarea is adjusted as discussed herein. FIG. 2f illustrates the desiredsegment.

[0022] Below are provided the mathematical details of the process.

[0023] BOX A

[0024] INPUT: RGB image I(x,y).

[0025] OUTPUT: Approximate CIELAB image {circumflex over (F)}(I(x,y)).

[0026] PROCESS:

[0027] Let the input RGB image be given by I(x,y), where x and y are thespatial location in the image. If the image is contained in a standardRGB color space (e.g. sRGB See M. Anderson et. al. , “Proposal for aStandard Default Color Space for the Internet,” IS& /SID 4th ColorImaging Conference, pp. 238-246, 1996.), then the conversion from RGB toCIELAB is well defined. If the image is not contained in a standard RGBcolor space then it is necessary to determine a transformation from RGBto CIELAB. This transformation only needs to be determined one time, butis applied to every captured image. Specifically, the transformation canbe determined as follows:

[0028] 1. The mapping from RGB to CIEXYZ for the input device (likely aweb camera or a digital camera) is modeled as a linear system

t=G ^(T) c

[0029]  where the RGB pixel value is given by the 3-element vector c, tis the 3-element CIEXYZ value, and G is a 3×3 matrix that is determinedas outlined in step 2.

[0030] 2. The matrix G is determined by measuring a set of N sampleswith the camera. The CIELAB values of the N samples are determined witha colorimeter. A matrix G is then determined by solving$G = {\arg \left( {\min\limits_{H}\left( {\frac{1}{N}{\sum\limits_{i = 1}^{N}{{{F\left( {H^{T}c_{i}} \right)} - u_{i}}}}} \right)} \right)}$

[0031]  via a modified Newton method where the CIELAB values are givenby the vector sequence {u_(i)}_(i=1) ^(N), the measured values from thecamera (determined by averaging across a set of pixels) are given by{c_(i)}_(i=1) ^(N), and the mapping from CIEXYZ to CIELAB is given bythe function F. See G. Wyszecki, W. S. Stiles, Color Science: Conceptsand Methods, Quantitative Data and Formaulae, John Wiley & Sons, NY,1982, pp. 166-169. For example, for a Nikon CoolPix 990 camera, G is:[0.2818 0.1444 0.0653 0.1803 0.2872 0.0382 0.0404 0.0131 0.3647]

[0032] 3. For simplicity denote the output of this box as {circumflexover (F)}(I(x,y)).

[0033] BOX B

[0034] INPUT: Approximate CIELAB image {circumflex over (F)}(I(x,y)).

[0035] OUTPUT: K-Means segmented image K(x,y).

[0036] The output CIELAB image {circumflex over (F)}(I(x,y)) is providedas input to an iterative K-Means algorithm in Box B. Specifically thealgorithm is as follows:

[0037] 1. The bands of the image {circumflex over (F)}(I(x,y)) (i.e. the3 color planes (LAB channels) of the image) may be equalized or scaledto provide increased dynamic range, and to maximize the likelihood ofbeing able to differentiate between desired and undesired features. Theideal scaling values are determined experimentally, by testing variousscaling values on typical images for a given application.

[0038] 2. An initial set of K vector values, k_(l) l=1, . . . , K isselected that may likely differentiate between the desired feature andundesired features in the image. These values should have beendetermined through experimentation, which involves testing variousvalues on typical images.

[0039] 3. Each pixel is assigned to one of the K classes. Which class apixel d is in is determined using

[0040]${c = {{{\arg \left( {\min\limits_{l}\left( {{d - k_{l}}} \right)} \right)}\quad l} = 1}},\quad \ldots \quad,K$

[0041]  where k_(l) is the value of the lth class, and k_(c) is theclass to which pixel d is assigned. In other words, pixel d is assignedto the class closest to it in terms of Euclidean distance.

[0042] 4. After each pixel has been assigned to a class, update orrefine the class values using

k _(l) =E{[d|d∈k _(l)]}

[0043]  which are simply the class value means and reassign the pixelsbased on the refined class values.

[0044] 5. If the maximum number of iterations has been achieved (anumber that is determined experimentally) or no pixels have changedclasses, then stop. If not, then go to step 3.

[0045] BOX C

[0046] INPUT: Approximate CIELAB Image I(x,y).

[0047] OUTPUT: Blurred Image B(I(x,y))

[0048] PROCESS: See W. K. Pratt, Digital Image Processing, John Wiley &Sons, NY, 1991, pp171-191.

[0049] The image I(x,y) is blurred in Box C using a convolution process.Mathematically, convolution is given byB(I(x, y)) = ∫_(−∞)^(∞)∫_(−∞)^(∞)H(a, b)I(x − a, y, b)  a  b

[0050]  where H(a,b) is the blur kernel. The size of the blur kernelshould be smaller than the feature that is being detected, and the shapeshould be symmetric. An example of a symmetric H is given by theequation below, where N is the radius of the blur kernal:

H(a,b)=1/(2N+1)² ∀−N≦a≦N & −N≦b≦N

[0051] BOX D

[0052] INPUT: Blurred Image B(I(x,y))

[0053] OUTPUT: Pixel Location [x₀,y₀]

[0054] PROCESS:

[0055] In Box D, a pixel location that is likely in the desired featureis determined. If the color of the expected feature value is given bythe three element vector v, then the following algorithm can be used toidentify the pixel$\left\lbrack {x_{0},y_{0}} \right\rbrack = {\arg \left( {\min\limits_{\lbrack{a,b}\rbrack}\left( {{{B\left( {I\left( {a,b} \right)} \right)} - v}}^{2} \right)} \right)}$

[0056]  that is simply to find the pixel in the blurred image that isclosest to v. The solution to the above optimization problem is achievedby an exhaustive search.

[0057] BOX E

[0058] INPUT: Pixel Location [x₀,y₀]& K-means image K(i,j).

[0059] OUTPUT: Binary Mask Image M(x,y)

[0060] PROCESS:

[0061] Box E uses the output of the K-Means algorithm (Box B) along withthe value [x₀,y₀](Box D) to create a mask of the desired image feature.The algorithm is as follows:

[0062] 1. Let there be R rows and C columns in the image.

[0063] 2. Create a pixel connection definition. One definition that isuseful is an 8-point connection. In this case, a pixel d is consideredconnected to every pixel surrounding it. Mathematically, if d is atlocation [x,y], then d is connected to the pixels at locations {[x1,y1],[x−1,y], [x−1,y+1], [x,y−1], [x,y+1], [x+1,y−1], [x+1,y], [x+1,y+1]}.

[0064] 3. Create a mask image M(x,y) of size R×C that is zeroeverywhere.

[0065] 4. Assign the pixel location [x₀,y₀]with a value 1.

[0066] 5. Create a temporary mask image M_(T)(x,y), which is the same asM(x,y).

[0067] 6. Starting from the spatial locations {[0,0],[R,0],[0,C],[R,C]},run through the pixels in the images (i.e. step through the pixels in avariety of ways). For a pixel at location [p,q], assign a value 1 toM_(T)(p,q) if the following conditions are satisfied:

[0068] Pixel M(p,q) is connected to a pixel with value 1.

[0069] Pixel K(p,q) is in the same class as K(x₀,y₀).

[0070] 7. Compare the temporary mask M_(T)(x,y) to the mask M(x,y) .Test if any pixels have been reassigned. If no pixels were reassigned,then stop. Otherwise, continue to step 8.

[0071] 8. Copy the temporary mask M_(T)(x,y) to the mask M(x,y) and goto step 6.

[0072] BOX F

[0073] INPUT: Binary Mask Image M(x,y)

[0074] OUTPUT: Decision—Was segmentation successful?

[0075] PROCESS:

[0076] A successful segmentation occurs if the feature is within abounding box.

[0077] If the mask reaches the edge of this box, then the segmentationalgorithm failed.

[0078] If the mask does not reach the edge of this box, then thesegmentation was successful.

[0079] BOX G

[0080] INPUT: Decision from Box F, Mask Image M(x,y) & Input ImageI(x,y).

[0081] OUTPUT: Image containing only the segmented feature J(x,y).

[0082] PROCESS:

[0083] If the output of Box E, M(x,y) is within a pre-defined boundary(tested in Box F), then it is assumed that the desired feature wasfound. In this case, the mask M(x,y) should be applied to the inputimage I(x,y)through a point by point operation. Mathematically, thisprocess is: ${J\left( {x,y} \right)} = \left\{ \begin{matrix}{I\left( {x,y} \right)} & {{{for}\quad {M\left( {x,y} \right)}} = 1} \\0 & {{{for}\quad {M\left( {x,y} \right)}} \neq 1}\end{matrix} \right.$

[0084] BOX H

[0085] INPUT: Decision from Box F

[0086] OUTPUT: Restart Image Capture

[0087] PROCESS:

[0088] If M(x,y) is outside the pre-defined boundary, then the algorithmfailed and a new image I(x,y) is captured and the process repeats.

[0089] The invention is illustrated in more detail by the followingnon-limiting example.

EXAMPLE

[0090] An image is shown in FIG. 2. FIG. 2a shows a boxed region whichindicates the area of interest. FIG. 2b illustrates the output from aK-Means Algorithm in CIELAB space. In this example, 5 classes (K=5) wereused. The selected in the coat are clearly selected (they are dark bluehere). This image is the typical output of a K-means algorithm. This isthe output from BOX B in the flow chart diagram. In this example theobject is to select one green portion of the jacket. Within the boundingbox, several green portions exist, but the target is the largestconnected portion. To determine a pixel in the largest portion, theimage is blurred in the bounding box. This is the output from BOX C inthe flow chart diagram. The greenest pixel in the blurred image isselected by the X as illustrated. This pixel location is the output fromBOX D in the flow chart diagram. The mask image is created from theK-means image and the position marked by the X using the algorithmdescribed for BOX E. The result is then checked to determine if the maskis at the edges of the bounding box. If not, the feature is complete.This is the analysis made by BOX F. In this case, the mask is applied tothe input RGB image to obtain the segmented image. This is the outputfrom BOX G. Once the segmentation process is completed, a correction canbe applied to the segmented portion (e.g. re-color the teeth to whitenthem). The specific nature and amount of correction is predetermined asa function of the process being simulated (e.g. the degree to whichteeth can be whitened). The guidelines for the correction can becaptured as an equation, algorithm, or look-up table which can be, forexample, used with interpolation to map current colors to new colors.Once the correction is applied and the new pixels are determined, theold pixels are replaced with the modified ones.

[0091] The segmentation method can be used in a device in which variouscomponents are integrated into a cabinet 10 that is designed to hang ona wall or sit on a shelf or counter. The device includes a means forcapturing a digital image—typically either a conventional web cam ordigital camera 14, a display such as a conventional LCD-type colordisplay 12, a source of light (not shown), and an internal processorthat runs the subject software. In its idle state, the display screencan display introductory information (marketing-oriented messages,instructions, or continuous video of images as customers or individualsmay see as they pass the camera), and invites the customer to activatethe unit by pressing the “go” button 16 on the front of the unit. The gobutton activates the software. A live image of the subject—as capturedby the camera—appears on the screen. The subject is instructed (using onscreen prompts) to position himself or herself such that the area ofinterest (teeth or hair, for example) appears in the on-screen box 18,and to press the “go” button 16 again. Pressing the go button the secondtime freezes the image on the screen, and begins the segmentation andcolor analysis. The device segments the image, identifying the pixels ofinterest, measures the color at multiple points, and calculates anaverage color value. The device performs calculations based on theinitial coloring and displays the result, in this case an image of thecustomer having whiter teeth or an alternative hair color. After anappropriate delay, the device returns to the idle state.

[0092] Having described the invention in detail and by reference tospecific embodiments thereof, it will be apparent the numerousmodifications and variations are possible without separating from thespirit and scope of the invention.

What is claimed is:
 1. A method for isolating an element of an image made up of pixels comprising the steps of classifying the pixels into different groups based on the color value of the pixel, blurring the image, locating a pixel in the blurred image that has a predetermined color value corresponding to the element to be isolated, and growing a mask from the located pixel.
 2. The method of claim 1 wherein the step of classifying the pixels into groups is performed using a K-means algorithm.
 3. The method of claim 2 wherein the step of growing the mask is performed by adding to the located pixel the pixels in the group in which the located pixel is classified that are spatially connected to the located pixel using a connection algorithm.
 4. The method of claim 3 wherein the color of a pixel in the mask is adjusted.
 5. The method of claim 4 wherein prior to the step of classifying the image, the image is in RGB color space and is converted to the CIELAB color space.
 6. The method of claim 5 wherein the step of converting the image to the CIELAB color space is modeled as a linear system t=G ^(T) c where the pixel value is given by the 3-element vector c, t is the 3-element CIEXYZ value, and G is a 3×3 matrix that is determined by solving $G = {\arg \left( {\min\limits_{H}\left( {\frac{1}{N}{\sum\limits_{i = 1}^{N}{{{F\left( {H^{T}c_{i}} \right)} - u_{i}}}}} \right)} \right)}$

where the CIELAB values are given by the vector sequence {u_(i)}_(i=1) ^(N), the RGB values are given by {c_(i)}_(i=1) ^(N), and the mapping from CIEXYZ to CIELAB is given by the function F.
 7. The method of claim 6 wherein in the step of classifying the pixels includes increasing the dynamic range of the image.
 8. The method of claim 7 wherein the image is blurred by using a convolution process including a blur kernel and the blur kernel is smaller than the feature to be isolated.
 9. The method of claim 8 wherein the convolution process is given by the equation B(I(x, y)) = ∫_(−∞)^(∞)∫_(−∞)^(∞)H(a, b)I(x − a, y, b)  a  b

where H(a,b) is the blur kernel.
 10. The method of claim 9 wherein the function H is given by the equation H(a,b)=1/(2N+1)² ∀−N≦a≦N & −N≦b≦N where N is the radius of the blur kernel.
 11. The method of claim 9 wherein the step of locating the pixel uses the algorithm $\left\lbrack {x_{0},y_{0}} \right\rbrack = {\arg \left( {\min\limits_{\lbrack{a,b}\rbrack}\left( {{{B\left( {I\left( {a,b} \right)} \right)} - v}}^{2} \right)} \right)}$

where the color of the expected feature value is given by the three element vector v.
 12. The method of claim 2 is wherein the number of groups is 3 to
 5. 13. The method of claim 2 wherein the number of groups is sufficient to differentiate the element to be isolated from other elements of the image.
 14. The method of claim 2 wherein the step of classifying the pixels includes refining the color value of the different groups based on the group average and reclassifying the pixels in the image based on the refined color values.
 15. A computer-readable medium containing instructions for controlling a processor to isolate an element in an image made up of pixels by a method comprising the steps of classifying the pixels into different groups based on the color value of the pixel, blurring the image, locating a pixel in the blurred image that has a predetermined color value corresponding to the element to be isolated, and growing a mask from the located pixel.
 16. The medium of claim 15 wherein the step of classifying the pixels into groups is performed using a K-means algorithm.
 17. The medium of claim 16 wherein the step of growing the mask is performed by adding to the located pixel the pixels in the group in which the located pixel is classified that are spatially connected to the located pixel using a connection algorithm.
 18. The medium of claim 16 wherein there are 3 to 5 groups.
 19. The medium of claim 17 wherein the color of a pixel in the mask is adjusted.
 20. The medium of claim 19 wherein prior to the step of classifying the image, the image is in RGB color space and is converted to the CIELAB color space.
 21. An apparatus for displaying an image comprising: a camera for sensing an image and placing data corresponding to the image in memory, a processor or logic circuit programmed to isolate an element in an image made up of pixels and modifying the isolated element by a method including the steps of classifying the pixels into different groups based on the color of the pixel, blurring the image, locating a pixel in the blurred image that has predetermined color value corresponding to the element to be isolated, growing a mask from the located pixel, the mask corresponding to the isolated element, and modifying the color of the pixels in isolated element; and a display for displaying the image including the modified isolated element.
 22. The apparatus of claim 21 wherein the display includes a target area for locating the element in the image. 